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Abstract 

An analytical formula for the occurence probability of Markovian stochastic 
paths with repeatedly visited and/or equal departure rates is derived. This 
formula is essential for an efficient investigation of the trajectories belonging 
to random walk models and for a numerical evaluation of the 'contracted 
path integral solution' of the discrete master equation [Phys. Lett. A 195, 
128 (1994)]. 
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I. INTRODUCTION 



Stochastic processes play an important role in all scientific fields dealing with systems 
that are subject to inherent or external random infiuences ('fiuctuations'). Therefore, numer- 
ous methods and models have been developed for the description of stochastically behaving 
systems HTj-Q. The fields of application reach from physics over chemistry [Q,^ and 
biology to the economic and social sciences [^,01 • 

In this paper we will focus on Markovian stochastic processes which do not essentially 
depend on so-called memory effects so that correlations of present transitions with past states 
of the considered system (except the last one) can be neglected. We can distinguish cross- 
section oriented methods describing the temporal evolution of the distribution of states and 
longitudinal- data oriented methods delineating single stochastic trajectories (time series). If 
one is confronted with a continuous state space the distribution of states is governed by the 
Fokker-Planck equation and corresponding trajectories follow the Langevin equation 

(stochastic differential equation) Related methods exist for quantum mechanical 

systems 0. In the following we will concentrate on systems with a discrete state space. 
Then, the distribution of states satisfies the master equation |l|J^ whereas the corresponding 
time series are determined by random walk models and generated by means of Monte- 
Carlo simulations . 

In a recent paper an important relation between the distribution of states and the 
occurence probabilities of paths has been established which was called the 'contracted path- 
integral solution'. However, during its numerical implementation, this relation turned out to 
be restricted to the infrequent case of pure birth processes (uni-directional transitions), since 
it did not provide an analytical formula for paths with repeatedly visited states. Therefore, 
this paper presents the non-trivial derivation of the missing occurence-probability formula 
for 'degenerate paths' (Sec. III). 

The numerical implementation of the 'contracted path-integral solution' is outlined in 



Section IV. Since the 'breadth-first' procedure |T^ is very inefficient with respect to computer 
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time and memory (even for a few system states only), our algorithm bases on the 'depth-first' 
procedure [|12| . The suitability and correctness of this new numerical method is illustrated 
by an example concerning Brownian motion. 

Section V summarizes the results of the paper and discusses further fields of application. 



Let A4 = {1, 2, ... , N} be the discrete set of possible states of the considered stochasti- 
cally behaving system. Moreover, let w{j\i) represent the transition rate (i.e. the transition 
probability per unit time) for state changes from state i E Ai to state j 7^ i. In the fol- 
lowing we will assume that w{j\i) is time-independent. Then, the master equation of the 
corresponding Markovian stochastic process reads 



where P{i,t) denotes the probability of the system to be in state i at the time t. 

Alternatively, we can consider the associated random walk. Let iq be the state of the 
system at the initial time to, and let ti denote the transition times at which the system 
changes its state from ii_i to ii. The corresponding stochastic time series up to time t with 

tn ^ t tn+1 IS 



and can be numerically generated by means of a Monte- Carlo simulation method ||1CI|| . 

If we are interested in the statistical properties of longitudinal (trajectory-related) quan- 
tities belonging to the considered stochastic process, we need to evaluate a large number 
of time series with respect to certain characteristics. This is connected with a considerable 
computational effort. However, if one is not interested in the respective times ti at which 
the single transitions take place, but only in the path (sequence of states) which the system 
takes, this effort can be very much simplified. 



II. OCCURENCE PROBABILITIES OF PATHS 




(1) 




(2) 
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In order to illustrate this, let 

Cn:= io ii in (3) 

be the path which the system takes up to the time t. Note that w{ii\ii-i)dTi^i is the 
probability of changing from state to ii between time ti and ti + (ir;_i and that e"'^''^' 
with the departure rate (overall transition rate) 

N 

wi := ^iA^i) (4) 
1=1 

is the (survival) probability with which the system stays in state ii for a time interval (the 
survival time) Ti := ti^i — ti > {rn := t — tn). Therefore, multiplying the probability 
P{iQ,tQ) of the initial state io with the survival probabilities as well as the probabilities of 
the transitions involved and, afterwards, integrating with respect to the possible survival 
times Ti, we obtain the occurence probability P{Cn, t) with which the system takes the path 
Cn up to the time t := to + r ||13| : 



P(C„,r) = e-'""^'^ 1 dTn-iw{in\in-i)e-'"-'^-' . . . ldTow{ii\io)e-^''^'5[j2ri - r\p{io,to) . 

^'=0 ^ 

(5) 

Here, the (5-function guarantees that the survival times sum up to the available time r. 
Inserting its Laplace-representation 

— c+ioo 



' -c-ioo 



. \i=o ) 



(6) 



with a suitable constant c > and carrying out the integrations with respect to the survival 
times Ti, we finally obtain the formula ||13|,|ll],Q 



-C+lOO 



1 /■ e"""^ 
P(C„,r) = — y du— w(C„)P(io,to) , (7) 

-c-ioo ni"?^/ — U) 



i=0 

where we have introduced the abbreviation 
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Siio if n = 

<Cn):={n (8) 

[[w[ii\ii-i) if n > 1 
1=1 

with the Kronecker function Sij. 

It is obviously much easier to calculate formula (|^) than to count and evaluate a huge 
number of stochastic trajectories. However, for this calculation we must first evaluate the 
integral by means of the residue theorem. If all departure rates wi are identical (i.e. wi = w), 
we get 

P(C„, r) = ^e-"'^«;(C„)P(^o, to) , (9) 

whereas we obtain 

n 

P{Cn, r) = Y^ fn{Wk, T)w{Cn)P{io, to) (lO) 
k=0 

with 

fn{u,T):= „ ^ (11) 

n i^i-u) 

1=0 

if the departure rates wi are pairwise different from each other. However, serious problems 
arise in deriving a general result for cases where different departure rates wi occur, but some 
of them multiple times. 



III. DEGENERATE PATHS 

In the following, we will call paths for which some departure rate wi occurs twice or more 
often degenerate paths. Note that all paths with repeatedly visited states are degenerate! 

For this reason let mi = m{wi) be the multiplicity of the departure rate wi in path C„ 
(/ G {0, 1, . . . ,n}). Then we have the relation Yl,wi = + 1, and formula (|^ can be 
rewritten in the form 
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P{Cn,T) 



1 

27ri 



— P(«o,to) 



(12) 



where we have changed the integration direction. Applying the residue theorem, we obtain 



P(C„,r) = -^ 



t(7(C„)P(iio,to) 



U=Wk 



where fj{^\u,T) := d"^fn{u,T)/du'^. Introducing the functions 



we can represent (^) in the form 



fn{u,T)ml 



P{Cn,T) = J2Pmkil^k,r)fniWk,T)w{Cn)P{io,to) 
k=0 



(13) 



(14) 



(15) 



For Pm{u, t) we can derive the recursive relation 



Pm[U,T) 



-1 d 

mfn{u,T) du 

-1 



fn{u,T) 



{-ly 



fniu,T){m - 1)! 



(m-2). 



m 



" 1 \ d 

E ^-rjp_,(.,r) + -p_,(«,r) 



(16) 



This shows that Pm{u, t) is a polynomial in r of order m — 1 since a comparison with ([lOD 
provides us with pi(u, r) = 1. However, formula (|16D is not very suitable for a numerical 
implementation or a derivation of the explicit form of Pm{u,T). In order to evaluate (|14|), 
we try to find the (m — l)st derivative Z^™""^-' of /„. Since we would not succeed with the 
usual procedure, we need a trick and take the 'detour' over the first derivative 

1 



(17) 



The {no — l)st derivative of this relation can now be determined as usual 

1 



ni=0 ^1- 



(18) 



Here, we have used the definitions ni := no — 1 — / and 

r):^l^(t^^-r).t - . (19) 

Formula ([18D obviously allows to express the derivatives f^'^\u^T) in terms of derivatives 
f^'^\u,T) of lower order ni < no- Therefore, we insert for fl^'^\u,T) again relation (|18|), 
etc. After a couple of iterations this procedure leads, for no > 1, to 

("0-1 ^(no-ni) / m-l (ni-n2) 
gMfiO)^J2 ^ + E ^ 
ni=l ^1 V n2=l ^2 

/ / ""0-1-1 (nno-i-n„o) X 

where we have separated the terms containing fl{^\u, r) = fn{u, t) from the rest and omitted 
the arguments {u, r) of the functions. In equation (pO|) we have to apply the convention 
J2n=i-^{---) '■= if /2 < h (as will be shown by Fig. 1). Consequently, the term with 
the n^oth derivative f^"°^ does not give a contribution to f^"'°\ since max(ni) = n-o — 1, 
max(r;,2) = ni — 1 = ng — 2, . . ., max(n„g) = no — n-o = 0. Therefore, we obtain the final 
result 

/ 1 \m+l / m-2 (m-l-ni) / m-l (ni-n2) 

Pm = V^T K'""^ + ^ K"^ + ^ 

for m > 2. The correctness of this formula is illustrated by Fig. 1. Fig 

IV. NUMERICAL EVALUATION OF THE 'CONTRACTED PATH-INTEGRAL 

SOLUTION' 



In Refs. |Tl|J^ it has been proved that the solution P{i,t) of master equation can be 
represented in the form 

oo oo N N N 

P(z,to + r) = EEm,r):=E E E ••• E Pi^o^^l^...^^n,r) 

(22) 



with in '■= i- According to (0), the probabihty P{i,to + r) to find state i at the time 
t = to + T is given as the sum over the occurence probabihties P{Cn, t) of all paths C„ which 
have an arbitrary length n but have led to state in = i within the time interval r. 

When (^2|) is evaluated numerically, one must restrict the summation to a finite number 
of relevant paths. Here, we will define a path C„ to be relevant if it fulfils the condition 

\r-{r)cj<a^^, (23) 

where {t)c„ '■= Z]fc=ol/^fc is the mean value of the occurence times of path C„ and 9c„ : = 
X]fc=o their variance (cf. |TT|,^). The parameter a is a measure for the accuracy of 

the above outlined approximation. According to our experience, a ^ 3 usually guarantees a 
reconstruction of 99% of the probability distribution P{i,t) which can be checked by means 
of the normalization condition J2iLi P{ht) = 1- 

An efficient implementation of the approximate 'contracted path-integral solution' on 
a serial computer bases on the path-search procedure 'depth- first' |]12[. After each step of 
this standard procedure which extends the previous path C„ by an additional state in+i 
(leading to the path C„+i := C„ in+i) the following quantities are calculated and stored 



in a so-called 'linked list' 12 



1 1 

{r)c„+-, ■■= {r)c„ + , Oc„^, := 9c„ + , w{Cn+i) := w{Cn)w{in+i\in) , 



(24) 



and 



fn+i{wk, r) := •^"^^^'^^ for k = 0,...,n with Wk 7^ Wn+i ■ (25) 

Wn+l - Wk 

If condition (^) is fulfilled, the occurence probability P{Cn, t) is calculated according to 
formula (p!5| ) and added to P{in+i, t). Afterwards, the next step of the 'depth-first' procedure 
is carried out. 



The path Cn+i is not further extended if (t)c„+i — a^9c„+-^ > r. This implies that 
the path C„+i and all longer paths C„+i in+2 — > . . . which include Cn+i as subpath are 
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irrelevant. Then, the procedure traces back its path Cn+i one step before it tries to extend 
the previous path C„ by another state i'^^^i — in+i + 1- 

Figure 2 illustrates the approximate 'contracted path-integral solution' for the example 
of Brownian motion which is characterized by nearest-neighbor transitions with the following 
transition rates: 

D if I?" - i| = 1 

wm = { (26) 
otherwise. 



Fig. 2 



V. SUMMARY AND FIELDS OF APPLICATION 

In this paper we were able to derive an analytical formula for the occurence probability 
of arbitrary paths including the complex case of repeatedly visited states and/or equal 
departure rates. This allows a numerical implementation of the 'contracted path-integral 
solution' of the discrete master equation. Other kinds of path-integral formalisms have 
been developed for chaotic mappings |]T^, the Schrddinger equation [|T3|, the Fokker-Planck 
equation [§,0, and also the master equation |[T^. 

The importance of the occurence-probability formula goes far beyond the new solution 
method for the master equation. It considerably simplifies the evaluation of the trajectories 
related with random walk models. Therefore, a simulation program for the calculation of the 
occurence probabilities of paths, the path-integral solution, most probable paths, and first- 
passage times has recently been developed at the University of Stuttgart. It is expected to 
be a useful tool for investigations in a number of current research fields concerning different 
types of random walks [|1^], noise-induced transitions [|18|, first-passage time problems |I9 



percolation PD[, critical behavior [pTf|, and diffusion in disordered or fractal media 
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FIGURES 
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FIG. 1. Comparison of the occurence probability P{Cn,T) calculated via formulas (fig), (11) 

10 (o) with the occurence probability of the same path 



and (21) for the case wi = w, mi = n + 1 
Cn according to (0) ( — ). 




0.0 



FIG. 2. Comparison of the approximate path-integral solution for Brownian motion (bars) with 
the numerical result of Runge-Kutta integration |23] for the corresponding master equation (grid). 
The diffusion coefficient was assumed to be D = 1, and the accuracy parameter was set to a = 3 
so that the path- integral method reconstructs about 99% of the exact probability distribution. 
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